Papers with subword regularization method
BPE-Dropout: Simple and Effective Subword Regularization (2020.acl-main)
Copied to clipboard
| Challenge: | Subword segmentation is widely used to address the open vocabulary problem in machine translation. |
| Approach: | They propose a method that stochastically corrupts the segmentation procedure of BPE and produces multiple segmentations within the same fixed BPE framework. |
| Outcome: | The proposed method produces multiple segmentations within the same fixed BPE framework. |
MaxMatch-Dropout: Subword Regularization for WordPiece (2022.coling-1)
Copied to clipboard
| Challenge: | Existing subword regularization methods are specialized to a particular tokenizer type. |
| Approach: | They propose a subword regularization method for WordPiece that uses a maximum matching algorithm for tokenization. |
| Outcome: | The proposed method improves the performance of text classification and machine translation tasks as well as other subword regularization methods. |